[CLI] Improve ray status for placement groups by rkooo567 · Pull Request #18289 · ray-project/ray

rkooo567 · 2021-09-01T22:18:35Z

Why are these changes needed?

For no node type, we format resources in a prettier way. It also displays how many of resources are reserved by pgs.
For usage, it includes usage / total of placement group. Previously, it only prints the "usage" of placement group, which was confusing

After

Node status
------------------------------------------------------------
 1 node(s) with resources:
  CPU: 3.0
  object_store_memory: 157286400.0
  GPU: 66.0 (66.0 reserved in placement groups)
  pg_custom: 66.0 (66.0 reserved in placement groups)
  node:192.168.1.3: 1.0
  memory: 8279552000.0

Resources
------------------------------------------------------------
Usage:
 0.0/3.0 CPU
 66.0/66.0 GPU (0.0/66.0 used by placement groups)
 0.00/7.711 GiB memory
 0.00/0.146 GiB object_store_memory
 66.0/66.0 pg_custom (0.0/66.0 used by placement groups)

Before

Node status
------------------------------------------------------------
 1 node(s) with resources: {'GPU_group_0_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_3_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_1_326f40a13dccec23e14b78fc80262f71': 1.0, 'GPU_group_3_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'pg_custom_group_9da53bf2a1710a4433453f8e072ef810': 5.0, 'pg_custom_group_6e150e681d33f39699685637ca0fdc5b': 5.0, 'bundle_group_3_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'GPU_group_0_4acd757a53c53f736b72800076076b53': 1.0, 'bundle_group_2_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_3_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_3_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_2_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'bundle_group_3_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'GPU_group_2_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_1_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'bundle_group_080c9d088f3049e7bb4139a4c9fdba1a': 5000.0, 'bundle_group_6e150e681d33f39699685637ca0fdc5b': 5000.0, 'bundle_group_4_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'pg_custom_group_efb63fec901c967f94cb97a3928d52cf': 5.0, 'pg_custom_group_e17ecaca21cc323f91f83f01a5d3f376': 5.0, 'GPU_group_4_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_1_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'GPU_group_e6314fb8a23d64a2df3b6c92a2d03821': 5.0, 'GPU_group_2_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_4_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'GPU_group_3_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_3_4acd757a53c53f736b72800076076b53': 1000.0, 'GPU_group_4_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_868294ac2d7a48c9447a16e7e915bd54': 5.0, 'GPU_group_0_d6975df5cdeda9b201fb62beb6311494': 1.0, 'pg_custom_group_4acd757a53c53f736b72800076076b53': 5.0, 'bundle_group_4_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'pg_custom_group_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'pg_custom_group_4_d6975df5cdeda9b201fb62beb6311494': 1.0, 'pg_custom_group_4_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_326f40a13dccec23e14b78fc80262f71': 5.0, 'GPU_group_0_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_1_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_4_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'pg_custom_group_1_4acd757a53c53f736b72800076076b53': 1.0, 'CPU': 3.0, 'bundle_group_4_326f40a13dccec23e14b78fc80262f71': 1000.0, 'bundle_group_1_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'GPU_group_4_efb63fec901c967f94cb97a3928d52cf': 1.0, 'pg_custom_group_3_326f40a13dccec23e14b78fc80262f71': 1.0, 'bundle_group_0_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'bundle_group_1_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_868294ac2d7a48c9447a16e7e915bd54': 5.0, 'GPU_group_3_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'bundle_group_4_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'pg_custom_group_0_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'GPU_group_d6975df5cdeda9b201fb62beb6311494': 5.0, 'pg_custom_group_0_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_4_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_1_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_2_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_e17ecaca21cc323f91f83f01a5d3f376': 5000.0, 'pg_custom_group_2_326f40a13dccec23e14b78fc80262f71': 1.0, 'pg_custom_group_3_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_0_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'GPU_group_76cfbe45659a1252fd4fe2f695ddd026': 5.0, 'pg_custom_group_1_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_1_326f40a13dccec23e14b78fc80262f71': 1000.0, 'GPU_group_0_efb63fec901c967f94cb97a3928d52cf': 1.0, 'GPU_group_4_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_1_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'pg_custom_group_d6975df5cdeda9b201fb62beb6311494': 5.0, 'GPU_group_1_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_3_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'pg_custom_group_0_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_2_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_080c9d088f3049e7bb4139a4c9fdba1a': 5.0, 'pg_custom_group_b47fe6e409dab4c127d34d05cf29017a': 5.0, 'object_store_memory': 157286400.0, 'bundle_group_2_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'GPU_group_3_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'pg_custom_group_1_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'pg_custom_group_2_efb63fec901c967f94cb97a3928d52cf': 1.0, 'pg_custom_group_2_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'bundle_group_d6975df5cdeda9b201fb62beb6311494': 5000.0, 'bundle_group_3_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'pg_custom_group_1_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'bundle_group_0_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'bundle_group_4_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'pg_custom_group_3_9da53bf2a1710a4433453f8e072ef810': 1.0, 'pg_custom_group_3_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_0_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'pg_custom_group_9d3bd7eb3e3d432729bccf4ae30568d7': 5.0, 'GPU': 66.0, 'pg_custom_group_1_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'pg_custom_group_2_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_1_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'bundle_group_3_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'pg_custom_group_2_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_326f40a13dccec23e14b78fc80262f71': 5.0, 'GPU_group_3_4acd757a53c53f736b72800076076b53': 1.0, 'GPU_group_4_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'GPU_group_1_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_0_efb63fec901c967f94cb97a3928d52cf': 1.0, 'pg_custom_group_2_d6975df5cdeda9b201fb62beb6311494': 1.0, 'pg_custom_group_4_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_3_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'bundle_group_4_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'pg_custom': 66.0, 'bundle_group_1_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'bundle_group_b47fe6e409dab4c127d34d05cf29017a': 5000.0, 'bundle_group_2_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'bundle_group_1_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'pg_custom_group_080c9d088f3049e7bb4139a4c9fdba1a': 5.0, 'bundle_group_efb63fec901c967f94cb97a3928d52cf': 5000.0, 'bundle_group_0_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'bundle_group_2_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'bundle_group_0_326f40a13dccec23e14b78fc80262f71': 1000.0, 'bundle_group_0_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'pg_custom_group_2_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'bundle_group_2_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'bundle_group_e6314fb8a23d64a2df3b6c92a2d03821': 5000.0, 'pg_custom_group_4_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_0_4acd757a53c53f736b72800076076b53': 1.0, 'GPU_group_4_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_26addd2b76ca2377438e18791ea0b8f4': 1000.0, 'GPU_group_2_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_1_26addd2b76ca2377438e18791ea0b8f4': 1000.0, 'GPU_group_1_326f40a13dccec23e14b78fc80262f71': 1.0, 'bundle_group_2_9da53bf2a1710a4433453f8e072ef810': 1000.0, 'pg_custom_group_3_6e150e681d33f39699685637ca0fdc5b': 1.0, 'bundle_group_1_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'GPU_group_2_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_0_326f40a13dccec23e14b78fc80262f71': 1.0, 'bundle_group_3_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'pg_custom_group_2_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_4_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_0_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_3_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_e6314fb8a23d64a2df3b6c92a2d03821': 5.0, 'GPU_group_1_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_2_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_1_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'pg_custom_group_4_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_0_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'bundle_group_2_efb63fec901c967f94cb97a3928d52cf': 1000.0, 'pg_custom_group_4_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_4_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'bundle_group_3_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'pg_custom_group_4_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_4_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'pg_custom_group_0_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_4_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'pg_custom_group_3_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_9da53bf2a1710a4433453f8e072ef810': 5.0, 'pg_custom_group_1_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_4_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'pg_custom_group_4_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_1_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_1_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_4_4acd757a53c53f736b72800076076b53': 1.0, 'GPU_group_2_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'bundle_group_4_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'bundle_group_3_326f40a13dccec23e14b78fc80262f71': 1000.0, 'GPU_group_4acd757a53c53f736b72800076076b53': 5.0, 'GPU_group_1_efb63fec901c967f94cb97a3928d52cf': 1.0, 'bundle_group_9da53bf2a1710a4433453f8e072ef810': 5000.0, 'pg_custom_group_1_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_0_080c9d088f3049e7bb4139a4c9fdba1a': 1000.0, 'GPU_group_efb63fec901c967f94cb97a3928d52cf': 5.0, 'pg_custom_group_3_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_0_d6975df5cdeda9b201fb62beb6311494': 1000.0, 'bundle_group_2_e6314fb8a23d64a2df3b6c92a2d03821': 1000.0, 'bundle_group_1_4acd757a53c53f736b72800076076b53': 1000.0, 'bundle_group_0_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'bundle_group_0_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'bundle_group_3_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'pg_custom_group_1_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'pg_custom_group_3_4acd757a53c53f736b72800076076b53': 1.0, 'pg_custom_group_76cfbe45659a1252fd4fe2f695ddd026': 5.0, 'GPU_group_1_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_0_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_2_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'bundle_group_2_868294ac2d7a48c9447a16e7e915bd54': 1000.0, 'GPU_group_2_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'GPU_group_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'bundle_group_2_326f40a13dccec23e14b78fc80262f71': 1000.0, 'bundle_group_4_b47fe6e409dab4c127d34d05cf29017a': 1000.0, 'bundle_group_2_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'pg_custom_group_2_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'pg_custom_group_2_4acd757a53c53f736b72800076076b53': 1.0, 'pg_custom_group_0_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_1_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_1_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_4_4acd757a53c53f736b72800076076b53': 1000.0, 'bundle_group_2_4acd757a53c53f736b72800076076b53': 1000.0, 'pg_custom_group_2_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'GPU_group_3_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_4acd757a53c53f736b72800076076b53': 5000.0, 'bundle_group_4_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'pg_custom_group_0_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_3_efb63fec901c967f94cb97a3928d52cf': 1.0, 'GPU_group_e17ecaca21cc323f91f83f01a5d3f376': 5.0, 'pg_custom_group_4_e17ecaca21cc323f91f83f01a5d3f376': 1.0, 'pg_custom_group_0_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_1_26addd2b76ca2377438e18791ea0b8f4': 1.0, 'GPU_group_2_326f40a13dccec23e14b78fc80262f71': 1.0, 'node:192.168.1.3': 1.0, 'GPU_group_4_4acd757a53c53f736b72800076076b53': 1.0, 'bundle_group_1_e17ecaca21cc323f91f83f01a5d3f376': 1000.0, 'GPU_group_4_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_1_4acd757a53c53f736b72800076076b53': 1.0, 'memory': 8279552000.0, 'bundle_group_4_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'pg_custom_group_2_6e150e681d33f39699685637ca0fdc5b': 1.0, 'pg_custom_group_2_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'GPU_group_2_4acd757a53c53f736b72800076076b53': 1.0, 'pg_custom_group_3_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'GPU_group_1_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'bundle_group_0_4acd757a53c53f736b72800076076b53': 1000.0, 'pg_custom_group_3_d6975df5cdeda9b201fb62beb6311494': 1.0, 'GPU_group_3_e6314fb8a23d64a2df3b6c92a2d03821': 1.0, 'pg_custom_group_3_b47fe6e409dab4c127d34d05cf29017a': 1.0, 'GPU_group_0_9d3bd7eb3e3d432729bccf4ae30568d7': 1.0, 'GPU_group_1_9da53bf2a1710a4433453f8e072ef810': 1.0, 'pg_custom_group_4_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'bundle_group_3_76cfbe45659a1252fd4fe2f695ddd026': 1000.0, 'pg_custom_group_0_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'pg_custom_group_0_326f40a13dccec23e14b78fc80262f71': 1.0, 'GPU_group_0_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_0_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'GPU_group_b47fe6e409dab4c127d34d05cf29017a': 5.0, 'GPU_group_1_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_3_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_9d3bd7eb3e3d432729bccf4ae30568d7': 5.0, 'bundle_group_1_6e150e681d33f39699685637ca0fdc5b': 1000.0, 'GPU_group_4_9da53bf2a1710a4433453f8e072ef810': 1.0, 'bundle_group_868294ac2d7a48c9447a16e7e915bd54': 5000.0, 'bundle_group_326f40a13dccec23e14b78fc80262f71': 5000.0, 'GPU_group_6e150e681d33f39699685637ca0fdc5b': 5.0, 'pg_custom_group_0_d6975df5cdeda9b201fb62beb6311494': 1.0, 'bundle_group_3_9d3bd7eb3e3d432729bccf4ae30568d7': 1000.0, 'GPU_group_2_9da53bf2a1710a4433453f8e072ef810': 1.0, 'GPU_group_3_6e150e681d33f39699685637ca0fdc5b': 1.0, 'GPU_group_0_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'GPU_group_0_76cfbe45659a1252fd4fe2f695ddd026': 1.0, 'GPU_group_1_868294ac2d7a48c9447a16e7e915bd54': 1.0, 'pg_custom_group_3_080c9d088f3049e7bb4139a4c9fdba1a': 1.0, 'bundle_group_9d3bd7eb3e3d432729bccf4ae30568d7': 5000.0, 'bundle_group_76cfbe45659a1252fd4fe2f695ddd026': 5000.0, 'GPU_group_0_e17ecaca21cc323f91f83f01a5d3f376': 1.0}

Resources
------------------------------------------------------------
Usage:
 0.0/3.0 CPU
 66.0/66.0 GPU
 0.00/7.711 GiB memory
 0.00/0.146 GiB object_store_memory
 66.0/66.0 pg_custom

Related issue number

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

rkooo567 · 2021-09-01T22:20:29Z

Btw, this is still confusing;

66.0/66.0 GPU (0.0/66.0 used by placement groups)

We uses 66 GPUs because they are all reserved by placement groups. I was thinking

0.0/0.0 GPU
0.0/66.0 GPU (reserved by placement groups)

this as an alternative format, but haven't implemented it because this can also be confusing...

ericl · 2021-09-02T18:45:28Z

test_cli failing

ericl · 2021-09-02T18:46:54Z

66.0/66.0 GPU (0.0/66.0 used by placement groups)

This doesn't grammatically make sense. Does it mean 66 is reserved in placement groups, but 0 is used within the group?

ericl

Comment

ericl

See comments

rkooo567 · 2021-09-02T23:38:17Z

@ericl

This doesn't grammatically make sense. Does it mean 66 is reserved in placement groups, but 0 is used within the group?

Yes. Right now the representation is

 66.0/66.0 GPU (X reserved in placement groups)

where X is "used" resources in placement groups.

and this will become

66.0/66.6 GPU (X/66.0 used in placement groups).

I agree this rep is a little weird. Another alternative is

Separate out resources used by pgs

0.0/0.0 GPU # Original resources
X/66.0 GPU (reserved by placement groups) # PG reserved resources

or

Use total reserved amount in the paranthesis and instead use the real usage

Y/66.0 GPU (66.0 reserved in placement groups) # Y == used (66) - pg_reserved (66) + pg_used (X)

One con of this option is that when your usages are mixed up with placement group usages, it is hard to read. Imagine

X CPU
Y Reserved
Z Used by pg

X + Z / 66.0 (Y reserved in placement groups) # but it is hard to know what are used by pg and what are not

ericl · 2021-09-03T00:36:37Z

I vote for (2)

…

On Thu, Sep 2, 2021, 4:38 PM SangBin Cho ***@***.***> wrote: @ericl <https://github.com/ericl> This doesn't grammatically make sense. Does it mean 66 is reserved in placement groups, but 0 is used within the group? Yes. Right now the representation is 66.0/66.0 GPU (X reserved in placement groups) where X is "used" resources in placement groups. and this will become 66.0/66.6 GPU (X/66.0 used in placement groups). I agree this rep is a little weird. Another alternative is 1. Separate out resources used by pgs 0.0/0.0 GPU # Original resources 0.0/66.0 GPU (reserved by placement groups) # PG reserved resources or 1. Use total reserved amount in the paranthesis and instead use the real usage X/66.0 GPU (66.0 reserved in placement groups) # X == used - pg_reserved + pg_used — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADUSSATLWMISU2EPJUJ4DUAADHLANCNFSM5DHQXLNA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

rkooo567 · 2021-09-03T06:18:27Z

@ericl @scv119 I'd like to get some final thoughts; I am thinking this;

66 CPU
56 pg reserved
4 used by CPU
40 used by pg reserved

# Current
60.0/66.0 (40 reserved in placement groups)

# New proposal
44.0/66.0 CPU (40 used, 56.0 reserved in placement groups )

Thought?

ericl · 2021-09-03T07:43:31Z

LGTM. The extra redundancy helps understanding.

…

On Thu, Sep 2, 2021, 11:18 PM SangBin Cho ***@***.***> wrote: @ericl <https://github.com/ericl> @scv119 <https://github.com/scv119> I'd like to get some final thoughts; I am thinking this; 66 CPU 56 pg reserved 4 used by CPU 40 used by pg reserved # Current 60.0/66.0 (40 reserved in placement groups) # New proposal 44.0/66.0 CPU (40 used, 56.0 reserved in placement groups ) Thought? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADUSROEWJQL5J5RJ4WVCLUABSD5ANCNFSM5DHQXLNA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

python/ray/tests/test_resource_demand_scheduler.py

Improve ray status

9231cb7

rkooo567 assigned scv119 and wuisawesome Sep 1, 2021

rkooo567 assigned ericl Sep 2, 2021

ericl approved these changes Sep 2, 2021

View reviewed changes

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 2, 2021

ericl reviewed Sep 2, 2021

View reviewed changes

ericl requested changes Sep 2, 2021

View reviewed changes

rkooo567 added 2 commits September 9, 2021 17:44

Merge branch 'master' into ray-status-pg-improve

3d57056

Add a new test

b0df649

rkooo567 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 10, 2021

rkooo567 requested a review from ericl September 10, 2021 01:34

remove

d3ed58e

ericl reviewed Sep 10, 2021

View reviewed changes

python/ray/tests/test_resource_demand_scheduler.py Outdated Show resolved Hide resolved

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 10, 2021

rkooo567 added 2 commits September 14, 2021 00:23

Merge branch 'master' into ray-status-pg-improve

7f731fb

Addressed code review.

27cb8be

rkooo567 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 14, 2021

rkooo567 requested a review from ericl September 14, 2021 17:10

ericl approved these changes Sep 14, 2021

View reviewed changes

ericl merged commit 31e1638 into ray-project:master Sep 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[CLI] Improve ray status for placement groups#18289

[CLI] Improve ray status for placement groups#18289
ericl merged 6 commits intoray-project:masterfrom
rkooo567:ray-status-pg-improve

rkooo567 commented Sep 1, 2021 •

edited

Loading

Uh oh!

rkooo567 commented Sep 1, 2021

Uh oh!

ericl commented Sep 2, 2021

Uh oh!

ericl commented Sep 2, 2021

Uh oh!

ericl left a comment

Uh oh!

ericl left a comment

Uh oh!

rkooo567 commented Sep 2, 2021 •

edited

Loading

Uh oh!

ericl commented Sep 3, 2021 via email

Uh oh!

rkooo567 commented Sep 3, 2021

Uh oh!

ericl commented Sep 3, 2021 via email

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

Conversation

rkooo567 commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

rkooo567 commented Sep 1, 2021

Uh oh!

ericl commented Sep 2, 2021

Uh oh!

ericl commented Sep 2, 2021

Uh oh!

ericl left a comment

Choose a reason for hiding this comment

Uh oh!

ericl left a comment

Choose a reason for hiding this comment

Uh oh!

rkooo567 commented Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericl commented Sep 3, 2021 via email

Uh oh!

rkooo567 commented Sep 3, 2021

Uh oh!

ericl commented Sep 3, 2021 via email

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rkooo567 commented Sep 1, 2021 •

edited

Loading

rkooo567 commented Sep 2, 2021 •

edited

Loading